Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 1: Dataset Exploration

Visualize the German Traffic Signs Dataset. This is open ended, some suggestions include: plotting traffic signs images, plotting the count of each sign, etc. Be creative!

The pickled data is a dictionary with 4 key/value pairs:

  • features -> the images pixel values, (width, height, channels)
  • labels -> the label of the traffic sign
  • sizes -> the original width and height of the image, (width, height)
  • coords -> coordinates of a bounding box around the sign in the image, (x1, y1, x2, y2). Based the original image (not the resized version).
In [1]:
# Load pickled data
import os
import pickle
import numpy as np
import tensorflow as tf
import cv2
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelBinarizer
from scipy.ndimage.interpolation import rotate
from random import randint

is_labels_encode = False

# TODO: fill this in based on where you saved the training and testing data
training_file = 'traffic-signs-data/train.p'
testing_file = 'traffic-signs-data/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
# ndarrays
X_data, y_data = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

print("Done loading")
Done loading

Data Visualization

In [6]:
unique_values, indices, counts = np.unique(y_data, return_index=True, return_counts=True)
for value, count in zip(unique_values, counts):
    print(value, count)
for index in indices:
    plt.imshow(X_data[index])
    plt.show()
0 210
1 2220
2 2250
3 1410
4 1980
5 1860
6 420
7 1440
8 1410
9 1470
10 2010
11 1320
12 2100
13 2160
14 780
15 630
16 420
17 1110
18 1200
19 210
20 360
21 330
22 390
23 510
24 270
25 1500
26 600
27 240
28 540
29 270
30 450
31 780
32 240
33 689
34 420
35 1200
36 390
37 210
38 2070
39 300
40 360
41 240
42 240

Create Validation Set

In [2]:
# Need to split validation set off first, prior to any data processing.
# Otherwise validation data will leak into the training set during the normalization process.
# Must shuffle training data prior to split!
# Will use 1/5 of current training set for validation set.  
# That would give us a 60/15/25 split which is on the low end for training size, but the autogenerated data will
# augment the training data, which should alleviate the concern of too small training size.

# Get randomized datasets for training and validation
X_train, X_validation, y_train, y_validation = train_test_split(
    X_data,
    y_data,
    test_size=0.2,
    random_state=832289)

print("Validation set created")
Validation set created

Data Augmentation

In [3]:
### Preprocess the data here.
### Feel free to use as many code cells as needed.

## Create horizontally flipped augmented data

## Create randomly rotated augmented data
## TODO: Determine if there's an efficient way to rotate each image by a random angle,
## rather than all images by same random angle.
X_rotate_1 = rotate(X_train, angle=randint(-45,45), axes=(1,2), reshape=False)
X_rotate_2 = rotate(X_train, angle=randint(-45,45), axes=(1,2), reshape=False)
X_rotate_3 = rotate(X_train, angle=randint(-45,45), axes=(1,2), reshape=False)
X_rotate_4 = rotate(X_train, angle=randint(-45,45), axes=(1,2), reshape=False)

## TODO: Zoom, Stretching, Color Perbutation

#orig_image = X_train[0]
#rotate_image = X_rotate_1[0]
#plt.imshow(orig_image)
#plt.show()
#plt.imshow(rotate_image)
#plt.show()

X_train = np.vstack((X_train, X_rotate_1, X_rotate_2, X_rotate_3, X_rotate_4))

y_train = np.concatenate((y_train, y_train, y_train, y_train, y_train))
y_train.shape
if(y_train[0] == y_train[31367]):
    print("Copying successful")
Copying successful

Data Preprocessing

In [4]:
# Normalization Method 1: Compute mean/stddev per pixel and channel across the entire batch.  
# Use these values to normalize
def normalize_dataset(data_set):
    return (data_set - np.mean(data_set, axis=0)) / (np.std(data_set, axis=0) + 1e-8)

# Normalization Method 2: Compute mean/stddev per image, normalize each image that way.
def normalize_dataset_alt(data_set):
    return (data_set - np.mean(data_set, axis=(1,2,3), keepdims=True)) / (np.std(data_set, axis=(1,2,3), keepdims=True) + 1e-8)

def normalize_grayscale(X):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    a = 0.1
    b = 0.9
    return a + ((X - X.min(axis=0))*(b-a))/(X.max(axis=0) - X.min(axis=0))

# Convert image to grayscale
def rgb2gray(X):
    X_gray = np.tensordot(X, [0.299, 0.587, 0.114], axes=([3],[0]))
    return normalize_grayscale(X_gray)
In [4]:
# Convert to grayscale
# X_train = rgb2gray(X_train)
# X_validation = rgb2gray(X_validation)
# X_test = rgb2gray(X_test)

# Reshape from (num_samples, 32, 32) -> (num_samples, 32, 32, 1)
# X_train = X_train[:,:,:, None]
# X_validation = X_validation[:,:,:, None]
# X_test = X_test[:,:,:, None]

# Normalize grayscale intensities
X_train = normalize_dataset_alt(X_train)
X_validation = normalize_dataset_alt(X_validation)
X_test = normalize_dataset_alt(X_test)



print(X_train.shape)
print("Done with normalization.")
(156835, 32, 32, 3)
Done with normalization.

OneHot Encode Labels

In [19]:
from sklearn.preprocessing import LabelBinarizer
if not is_labels_encode:
    # Turn labels into numbers and apply One-Hot Encoding
    encoder = LabelBinarizer()
    encoder.fit(y_train)
    y_train = encoder.transform(y_train)
    y_validation = encoder.transform(y_validation)
    y_test = encoder.transform(y_test)

    # Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
    y_train = y_train.astype(np.float32)
    y_validation = y_validation.astype(np.float32)
    y_test = y_test.astype(np.float32)
    is_labels_encode = True

print('Labels One-Hot Encoded')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-19-7f3d9e0a91bb> in <module>()
      1 from sklearn.preprocessing import LabelBinarizer
----> 2 if not is_labels_encode:
      3     # Turn labels into numbers and apply One-Hot Encoding
      4     encoder = LabelBinarizer()
      5     encoder.fit(y_train)

NameError: name 'is_labels_encode' is not defined

Save Processed Data

In [6]:
### Save Generated, Processed Data To Local File Cache
### Can't save entire training set due to bug, so splitting training data.
### http://stackoverflow.com/questions/31468117/python-3-can-pickle-handle-byte-objects-larger-than-4gb
directory = "processed_data/"

def saveData(pickle_filename, data):
    if not os.path.isfile(directory + pickle_filename):
        print('Saving data to pickle file...')
        try:
            with open(directory + pickle_filename, 'wb') as pfile:
                pickle.dump(data, pfile, pickle.HIGHEST_PROTOCOL)
        except Exception as e:
            print('Unable to save data to', pickle_filename, ':', e)
            raise
    print('Data cached in pickle file.')

for i, X_train_n in enumerate(np.split(X_train, 7)): 
    saveData("X_train" + str(i) + ".pickle", X_train_n)
saveData("y_train.pickle", y_train)
saveData('valid_test.pickle', {'X_validation': X_validation, 'X_test': X_test, 'y_validation': y_validation, 'y_test': y_test})
print("All data cached")
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
Saving data to pickle file...
Data cached in pickle file.
All data cached

Checkpoint

Start from here as augmented, preprocessed data is saved in local file.

In [5]:
import math
import pickle
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from tqdm import tqdm

directory = "processed_data/"
# Reload the data
def load_data(filename):
    with open(directory + filename, 'rb') as f:
        return pickle.load(f)

    
X_train = np.vstack([load_data('X_train' + str(i) + '.pickle') for i in range(7)])
y_train = load_data('y_train.pickle')
pickle_data = load_data('valid_test.pickle')
X_validation = pickle_data['X_validation']
y_validation = pickle_data['y_validation']
X_test = pickle_data['X_test']
y_test = pickle_data['y_test']
del pickle_data  # Free up memory

print('Data and modules loaded.')
Data and modules loaded.

Data Summary

In [6]:
### To start off let's do a basic data summary.

# number of training examples
n_train = len(y_train)

# number of validation examples
n_validation = len(y_validation)

# number of testing examples
n_test = len(y_test)

# what's the shape of an image?
image_shape = X_train[0].shape

# how many classes are in the dataset
n_classes = y_train.shape[1]

print("Number of training examples =", n_train)
print("Number of validation examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 156835
Number of validation examples = 7842
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Your model can be derived from a deep feedforward net or a deep convolutional network.
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

Model Creation

In [7]:
import tensorflow as tf

# Input [32 x 32 x 3] - Weight []
# Conv 1a [32 x 32 x 64] - Weight - [3 x 3 x 3 x 64]
# Conv 1b [32 x 32 x 64] - Weight - [3 x 3 x 64 x 64]
# Maxpool [16 x 16 x 64] - Weight - []
# Conv 2a [16 x 16 x 128] - Weight - [3 x 3 x 64 x 128]
# Conv 2b [16 x 16 x 128] - Weight - [3 x 3 x 128 x 128]
# Maxpool [8 x 8 x 128] - Weight - []
# Conv 3a [8 x 8 x 256] - Weight - [3 x 3 x 128 x 256]
# Conv 3b [8 x 8 x 256] - Weight - [3 x 3 x 256 x 256]
# Maxpool [4 x 4 x 256] - Weight - []
# FC 1 [1 x 1 x 512] - Weight - [4 x 4 x 256 x 512]
# Out [1 x 1 x 43] - Weight - [512 x 43]

# Total Weights = 3,263,680

layer_depth = {
    'layer_1': 64,
    'layer_2': 128,
    'layer_3': 256,
    'fully_connected': 512
}

weights = {
    'layer_1a': tf.Variable(tf.truncated_normal([3, 3, 3, layer_depth['layer_1']], stddev=1e-2)),
    'layer_1b': tf.Variable(tf.truncated_normal([3, 3, layer_depth['layer_1'], layer_depth['layer_1']], stddev=1e-2)),
    'layer_2a': tf.Variable(tf.truncated_normal([3, 3, layer_depth['layer_1'], layer_depth['layer_2']], stddev=1e-2)),
    'layer_2b': tf.Variable(tf.truncated_normal([3, 3, layer_depth['layer_2'], layer_depth['layer_2']], stddev=1e-2)),
    'layer_3a': tf.Variable(tf.truncated_normal([3, 3, layer_depth['layer_2'], layer_depth['layer_3']], stddev=1e-2)),
    'layer_3b': tf.Variable(tf.truncated_normal([3, 3, layer_depth['layer_3'], layer_depth['layer_3']], stddev=1e-2)),
    'fully_connected': tf.Variable(tf.truncated_normal([4*4*256, layer_depth['fully_connected']], stddev=1e-2)),
    'out': tf.Variable(tf.truncated_normal([layer_depth['fully_connected'], n_classes], stddev=1e-2))
}

biases = {
    'layer_1a': tf.Variable(tf.zeros(layer_depth['layer_1'])),
    'layer_1b': tf.Variable(tf.zeros(layer_depth['layer_1'])),
    'layer_2a': tf.Variable(tf.zeros(layer_depth['layer_2'])),
    'layer_2b': tf.Variable(tf.zeros(layer_depth['layer_2'])),
    'layer_3a': tf.Variable(tf.zeros(layer_depth['layer_3'])),
    'layer_3b': tf.Variable(tf.zeros(layer_depth['layer_3'])),
    'fully_connected': tf.Variable(tf.zeros(layer_depth['fully_connected'])),
    'out': tf.Variable(tf.zeros(n_classes))
}

# Dropout Probability
keep_prob = tf.placeholder(tf.float32)

def conv2d(x, W, b, strides=1):
    # Conv2D wrapper, with bias and relu activation
    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
    x = tf.nn.bias_add(x, b)
    return tf.nn.relu(x)

def maxpool2d(x, k=2):
    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1], padding='SAME')

def dropout(x, keep_probability):
    return tf.nn.dropout(x, keep_probability)

# Create model
def conv_net(x, weights, biases):
    # Layer 1 - 32*32*3 to 16*16*64
    conv1a = conv2d(x, weights['layer_1a'], biases['layer_1a'])
    conv1b = conv2d(conv1a, weights['layer_1b'], biases['layer_1b'])
    conv1 = maxpool2d(conv1b)
    conv1 = dropout(conv1, keep_prob)
    
    # Layer 2 - 16*16*64 to 8*8*128
    conv2a = conv2d(conv1, weights['layer_2a'], biases['layer_2a'])
    conv2b = conv2d(conv2a, weights['layer_2b'], biases['layer_2b'])
    conv2 = maxpool2d(conv2b)
    conv2 = dropout(conv2, keep_prob)

    # Layer 3 - 8*8*128 to 4*4*256
    conv3a = conv2d(conv2, weights['layer_3a'], biases['layer_3a'])
    conv3b = conv2d(conv3a, weights['layer_3b'], biases['layer_3b'])
    conv3 = maxpool2d(conv3b)
    conv3 = dropout(conv3, keep_prob)

    # Fully connected layer - 4*4*256 to 512
    # Reshape conv3 output to fit fully connected layer input
    fc1 = tf.reshape(
        conv3,
        [-1, weights['fully_connected'].get_shape().as_list()[0]])
    fc1 = tf.add(
        tf.matmul(fc1, weights['fully_connected']),
        biases['fully_connected'])
    fc1 = tf.nn.tanh(fc1)
    fc1 = dropout(fc1, keep_prob) 

    # Output Layer - class prediction - 512 to 10
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    return out

# Parameters
learning_rate = 0.0001
batch_size = 256
training_epochs = 100

# TF Graph input
x = tf.placeholder("float", [None, 32, 32, 3])
y = tf.placeholder("float", [None, n_classes])

# Output logits and predictions from net
logits = conv_net(x, weights, biases)
prediction = tf.nn.softmax(logits)

# Cross Entropy and Loss
cross_entropy = -tf.reduce_sum(y * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

# Accuracy
is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

# Optimizer
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)

# Initializing the variables
init = tf.initialize_all_variables()
In [6]:
def get_random_train_batch():
    train_perm = np.arange(500)
    np.random.shuffle(train_perm)
    X_train_acc = X_train[train_perm]
    y_train_acc = y_train[train_perm]
    return X_train_acc, y_train_acc

# Feed dicts for accuracy
valid_feed_dict = {x: X_validation, y: y_validation, keep_prob: 1.0}
test_feed_dict = {x: X_test, y: y_test, keep_prob: 1.0}

Training

In [10]:
# The accuracy measured against the validation set
best_validation_accuracy = 0.0

# Number of iterations of no validation accuracy improvement before stopping early
early_stop_num = 1000

# Last iteration that the validation accuracy improved
last_improvement_iteration = 0

total_iterations = 0

# Measurements use for graphing loss and accuracy
log_batch_step = 50
batches = []
train_acc_batch = []
valid_acc_batch = []

# Create Saver
saver = tf.train.Saver()
save_path = "model/best_model"

# Launch the graph
with tf.Session() as sess:
    sess.run(init)
    batch_count = int(math.ceil(n_train/batch_size))
    
    # Training cycle
    for epoch in range(training_epochs):
        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch+1, training_epochs), unit='batches')
        
        perm = np.arange(n_train)
        np.random.shuffle(perm)
        X_train = X_train[perm]
        y_train = y_train[perm]
        
        # Loop over all batches
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_X = X_train[batch_start : batch_start + batch_size]
            batch_y = y_train[batch_start : batch_start + batch_size]
            
            # Run optimization op (backprop) and cost op (to get loss value)
            _ = sess.run(optimizer, feed_dict={x: batch_X, y: batch_y, keep_prob: 0.5})
            
            total_iterations += 1
            
            # Log every 100 batches
            if not batch_i % log_batch_step:
                # Calculate Training and Validation accuracy
                X_train_acc, y_train_acc = get_random_train_batch()
                training_accuracy = sess.run(accuracy, feed_dict={x: X_train_acc, y: y_train_acc, keep_prob: 1.0})
                validation_accuracy = sess.run(accuracy, feed_dict=valid_feed_dict)
                
                # If validation accuracy is an improvement over best seen.
                if validation_accuracy > best_validation_accuracy:
                    
                    # Update best seen validation accuracy
                    best_validation_accuracy = validation_accuracy
                    
                    # Update last iteration to show validation accuracy improvement
                    last_improvement_iteration = total_iterations
                    
                    # Save all variables of the TensorFlow graph to file.
                    saver.save(sess=sess, save_path=save_path)
                    
                    print("Improved Validation Accuracy: ", validation_accuracy)
                
                # Log batches
                previous_batch = batches[-1] if batches else 0
                batches.append(log_batch_step + previous_batch)
                train_acc_batch.append(training_accuracy)
                valid_acc_batch.append(validation_accuracy)
            
            if total_iterations - last_improvement_iteration > early_stop_num:
                print("No improvement found in 1000 batches, stopping optimization.")
                break
            
    print("Optimization Finished!")

acc_plot = plt.subplot(212)
acc_plot.set_title('Accuracy')
acc_plot.plot(batches, train_acc_batch, 'r', label='Training Accuracy')
acc_plot.plot(batches, valid_acc_batch, 'b', label='Validation Accuracy')
acc_plot.set_ylim([0, 1.0])
acc_plot.set_xlim([batches[0], batches[-1]])
acc_plot.legend(loc=4)
plt.tight_layout()
plt.show()
    
    
# Improved Validation Accuracy:  0.996684
Epoch  1/100:   0%|          | 1/613 [01:35<16:11:10, 95.21s/batches]
Improved Validation Accuracy:  0.0502423
Epoch  1/100:   8%|▊         | 51/613 [06:09<3:40:51, 23.58s/batches]
Improved Validation Accuracy:  0.0549605
Epoch  1/100:  16%|█▋        | 101/613 [10:43<3:17:35, 23.16s/batches]
Improved Validation Accuracy:  0.0874777
Epoch  1/100:  25%|██▍       | 151/613 [14:58<2:45:54, 21.55s/batches]
Improved Validation Accuracy:  0.113236
Epoch  1/100:  33%|███▎      | 201/613 [19:10<2:25:32, 21.19s/batches]
Improved Validation Accuracy:  0.128666
Epoch  1/100:  41%|████      | 251/613 [23:19<2:04:02, 20.56s/batches]
Improved Validation Accuracy:  0.1886
Epoch  1/100:  49%|████▉     | 301/613 [27:29<1:47:59, 20.77s/batches]
Improved Validation Accuracy:  0.21015
Epoch  1/100:  57%|█████▋    | 351/613 [31:41<1:31:53, 21.04s/batches]
Improved Validation Accuracy:  0.232594
Epoch  1/100:  65%|██████▌   | 401/613 [35:53<1:15:11, 21.28s/batches]
Improved Validation Accuracy:  0.266514
Epoch  1/100:  74%|███████▎  | 451/613 [40:07<59:38, 22.09s/batches]
Improved Validation Accuracy:  0.317649
Epoch  1/100:  82%|████████▏ | 501/613 [44:22<41:29, 22.23s/batches]
Improved Validation Accuracy:  0.369676
Epoch  1/100:  90%|████████▉ | 551/613 [48:41<24:01, 23.25s/batches]
Improved Validation Accuracy:  0.406019
Epoch  1/100:  98%|█████████▊| 601/613 [53:03<04:49, 24.12s/batches]
Improved Validation Accuracy:  0.461872
Epoch  1/100: 100%|██████████| 613/613 [53:48<00:00,  3.75s/batches]
Epoch  2/100:   0%|          | 1/613 [02:00<20:30:44, 120.66s/batches]
Improved Validation Accuracy:  0.473349
Epoch  2/100:   8%|▊         | 51/613 [06:21<3:44:37, 23.98s/batches]
Improved Validation Accuracy:  0.495282
Epoch  2/100:  16%|█▋        | 101/613 [10:43<3:25:42, 24.11s/batches]
Improved Validation Accuracy:  0.539658
Epoch  2/100:  25%|██▍       | 151/613 [15:04<3:04:41, 23.99s/batches]
Improved Validation Accuracy:  0.560699
Epoch  2/100:  33%|███▎      | 201/613 [19:25<2:46:23, 24.23s/batches]
Improved Validation Accuracy:  0.596149
Epoch  2/100:  41%|████      | 251/613 [23:46<2:25:13, 24.07s/batches]
Improved Validation Accuracy:  0.616042
Epoch  2/100:  49%|████▉     | 301/613 [28:11<2:10:16, 25.05s/batches]
Improved Validation Accuracy:  0.646646
Epoch  2/100:  57%|█████▋    | 351/613 [32:35<1:49:15, 25.02s/batches]
Improved Validation Accuracy:  0.668962
Epoch  2/100:  65%|██████▌   | 401/613 [37:01<1:29:26, 25.31s/batches]
Improved Validation Accuracy:  0.678526
Epoch  2/100:  74%|███████▎  | 451/613 [41:25<1:07:08, 24.87s/batches]
Improved Validation Accuracy:  0.702117
Epoch  2/100:  82%|████████▏ | 501/613 [45:49<46:18, 24.80s/batches]
Improved Validation Accuracy:  0.721755
Epoch  2/100:  90%|████████▉ | 551/613 [50:14<25:51, 25.03s/batches]
Improved Validation Accuracy:  0.73897
Epoch  2/100:  98%|█████████▊| 601/613 [54:39<04:57, 24.77s/batches]
Improved Validation Accuracy:  0.751339
Epoch  2/100: 100%|██████████| 613/613 [55:24<00:00,  3.74s/batches]
Epoch  3/100:   0%|          | 1/613 [02:11<22:17:08, 131.09s/batches]
Improved Validation Accuracy:  0.75644
Epoch  3/100:   8%|▊         | 51/613 [06:36<3:54:30, 25.04s/batches]
Improved Validation Accuracy:  0.768681
Epoch  3/100:  16%|█▋        | 101/613 [11:01<3:37:06, 25.44s/batches]
Improved Validation Accuracy:  0.79138
Epoch  3/100:  25%|██▍       | 151/613 [15:28<3:17:24, 25.64s/batches]
Improved Validation Accuracy:  0.804004
Epoch  3/100:  33%|███▎      | 201/613 [19:52<2:51:03, 24.91s/batches]
Improved Validation Accuracy:  0.805662
Epoch  3/100:  41%|████      | 251/613 [24:15<2:28:38, 24.64s/batches]
Improved Validation Accuracy:  0.823004
Epoch  3/100:  49%|████▉     | 301/613 [28:37<2:07:06, 24.44s/batches]
Improved Validation Accuracy:  0.82989
Epoch  3/100:  57%|█████▋    | 351/613 [33:05<1:53:21, 25.96s/batches]
Improved Validation Accuracy:  0.839964
Epoch  3/100:  65%|██████▌   | 401/613 [37:32<1:30:04, 25.49s/batches]
Improved Validation Accuracy:  0.851951
Epoch  3/100:  74%|███████▎  | 451/613 [41:56<1:07:42, 25.08s/batches]
Improved Validation Accuracy:  0.857179
Epoch  3/100:  82%|████████▏ | 501/613 [46:24<48:11, 25.81s/batches]
Improved Validation Accuracy:  0.866871
Epoch  3/100:  90%|████████▉ | 551/613 [50:50<26:17, 25.45s/batches]
Improved Validation Accuracy:  0.872354
Epoch  3/100:  98%|█████████▊| 601/613 [55:13<04:53, 24.43s/batches]
Improved Validation Accuracy:  0.876945
Epoch  3/100: 100%|██████████| 613/613 [55:58<00:00,  3.75s/batches]
Epoch  4/100:   0%|          | 1/613 [01:59<20:19:09, 119.53s/batches]
Improved Validation Accuracy:  0.884978
Epoch  4/100:  16%|█▋        | 101/613 [10:48<3:42:28, 26.07s/batches]
Improved Validation Accuracy:  0.887401
Epoch  4/100:  25%|██▍       | 151/613 [15:09<3:05:46, 24.13s/batches]
Improved Validation Accuracy:  0.893012
Epoch  4/100:  33%|███▎      | 201/613 [19:31<2:46:01, 24.18s/batches]
Improved Validation Accuracy:  0.89722
Epoch  4/100:  41%|████      | 251/613 [23:54<2:29:14, 24.74s/batches]
Improved Validation Accuracy:  0.907294
Epoch  4/100:  57%|█████▋    | 351/613 [32:37<1:45:10, 24.08s/batches]
Improved Validation Accuracy:  0.910865
Epoch  4/100:  65%|██████▌   | 401/613 [37:02<1:28:29, 25.04s/batches]
Improved Validation Accuracy:  0.915838
Epoch  4/100:  74%|███████▎  | 451/613 [41:26<1:07:11, 24.89s/batches]
Improved Validation Accuracy:  0.920939
Epoch  4/100:  82%|████████▏ | 501/613 [45:48<45:39, 24.46s/batches]
Improved Validation Accuracy:  0.924637
Epoch  4/100:  98%|█████████▊| 601/613 [54:41<05:00, 25.04s/batches]
Improved Validation Accuracy:  0.929992
Epoch  4/100: 100%|██████████| 613/613 [55:26<00:00,  3.74s/batches]
Epoch  5/100:  16%|█▋        | 101/613 [11:00<3:40:13, 25.81s/batches]
Improved Validation Accuracy:  0.933945
Epoch  5/100:  33%|███▎      | 201/613 [19:49<2:58:01, 25.93s/batches]
Improved Validation Accuracy:  0.938409
Epoch  5/100:  41%|████      | 251/613 [24:21<2:45:10, 27.38s/batches]
Improved Validation Accuracy:  0.942489
Epoch  5/100:  57%|█████▋    | 351/613 [33:16<1:54:08, 26.14s/batches]
Improved Validation Accuracy:  0.945932
Epoch  5/100:  65%|██████▌   | 401/613 [37:42<1:29:55, 25.45s/batches]
Improved Validation Accuracy:  0.948355
Epoch  5/100:  74%|███████▎  | 451/613 [42:07<1:07:37, 25.05s/batches]
Improved Validation Accuracy:  0.95014
Epoch  5/100:  90%|████████▉ | 551/613 [50:50<25:11, 24.38s/batches]
Improved Validation Accuracy:  0.952053
Epoch  5/100:  98%|█████████▊| 601/613 [55:14<04:56, 24.67s/batches]
Improved Validation Accuracy:  0.953838
Epoch  5/100: 100%|██████████| 613/613 [55:59<00:00,  3.75s/batches]
Epoch  6/100:   0%|          | 1/613 [02:00<20:32:33, 120.84s/batches]
Improved Validation Accuracy:  0.954476
Epoch  6/100:   8%|▊         | 51/613 [06:26<3:57:48, 25.39s/batches]
Improved Validation Accuracy:  0.958174
Epoch  6/100:  25%|██▍       | 151/613 [15:10<3:11:04, 24.81s/batches]
Improved Validation Accuracy:  0.958812
Epoch  6/100:  41%|████      | 251/613 [23:59<2:35:18, 25.74s/batches]
Improved Validation Accuracy:  0.960597
Epoch  6/100:  49%|████▉     | 301/613 [28:22<2:07:51, 24.59s/batches]
Improved Validation Accuracy:  0.961362
Epoch  6/100:  57%|█████▋    | 351/613 [32:43<1:44:25, 23.91s/batches]
Improved Validation Accuracy:  0.96404
Epoch  6/100:  65%|██████▌   | 401/613 [37:09<1:30:47, 25.69s/batches]
Improved Validation Accuracy:  0.964422
Epoch  6/100:  82%|████████▏ | 501/613 [45:55<45:42, 24.49s/batches]
Improved Validation Accuracy:  0.965187
Epoch  6/100:  98%|█████████▊| 601/613 [54:38<04:53, 24.44s/batches]
Improved Validation Accuracy:  0.968758
Epoch  6/100: 100%|██████████| 613/613 [55:24<00:00,  3.75s/batches]
Epoch  7/100:   0%|          | 1/613 [02:04<21:12:24, 124.75s/batches]
Improved Validation Accuracy:  0.969651
Epoch  7/100:   8%|▊         | 51/613 [06:29<3:53:20, 24.91s/batches]
Improved Validation Accuracy:  0.970543
Epoch  7/100:  41%|████      | 251/613 [24:05<2:28:49, 24.67s/batches]
Improved Validation Accuracy:  0.971818
Epoch  7/100:  49%|████▉     | 301/613 [28:27<2:05:49, 24.20s/batches]
Improved Validation Accuracy:  0.972711
Epoch  7/100:  74%|███████▎  | 451/613 [41:32<1:05:27, 24.24s/batches]
Improved Validation Accuracy:  0.974241
Epoch  7/100:  98%|█████████▊| 601/613 [54:45<04:51, 24.31s/batches]
Improved Validation Accuracy:  0.974624
Epoch  7/100: 100%|██████████| 613/613 [55:31<00:00,  3.74s/batches]
Epoch  8/100:   0%|          | 1/613 [02:09<22:01:50, 129.59s/batches]
Improved Validation Accuracy:  0.976282
Epoch  8/100:   8%|▊         | 51/613 [06:30<3:43:11, 23.83s/batches]
Improved Validation Accuracy:  0.977174
Epoch  8/100:  16%|█▋        | 101/613 [10:55<3:36:56, 25.42s/batches]
Improved Validation Accuracy:  0.977302
Epoch  8/100:  25%|██▍       | 151/613 [15:19<3:12:03, 24.94s/batches]
Improved Validation Accuracy:  0.977684
Epoch  8/100:  33%|███▎      | 201/613 [19:42<2:51:31, 24.98s/batches]
Improved Validation Accuracy:  0.978704
Epoch  8/100:  57%|█████▋    | 351/613 [32:47<1:49:57, 25.18s/batches]
Improved Validation Accuracy:  0.979342
Epoch  8/100:  74%|███████▎  | 451/613 [41:35<1:04:29, 23.88s/batches]
Improved Validation Accuracy:  0.980872
Epoch  8/100:  82%|████████▏ | 501/613 [46:00<46:46, 25.06s/batches]
Improved Validation Accuracy:  0.981637
Epoch  8/100: 100%|██████████| 613/613 [55:25<00:00,  3.73s/batches]
Epoch  9/100:   0%|          | 1/613 [02:08<21:52:18, 128.66s/batches]
Improved Validation Accuracy:  0.983168
Epoch  9/100:  49%|████▉     | 301/613 [28:08<2:07:07, 24.45s/batches]
Improved Validation Accuracy:  0.983423
Epoch  9/100:  90%|████████▉ | 551/613 [49:54<25:00, 24.20s/batches]
Improved Validation Accuracy:  0.983678
Epoch  9/100: 100%|██████████| 613/613 [54:59<00:00,  3.74s/batches]
Epoch 10/100:   0%|          | 1/613 [02:00<20:30:21, 120.62s/batches]
Improved Validation Accuracy:  0.985208
Epoch 10/100:  25%|██▍       | 151/613 [15:00<3:06:12, 24.18s/batches]
Improved Validation Accuracy:  0.985718
Epoch 10/100:  49%|████▉     | 301/613 [28:05<2:04:46, 24.00s/batches]
Improved Validation Accuracy:  0.985973
Epoch 10/100:  57%|█████▋    | 351/613 [32:27<1:46:25, 24.37s/batches]
Improved Validation Accuracy:  0.986993
Epoch 10/100:  98%|█████████▊| 601/613 [54:09<04:46, 23.85s/batches]
Improved Validation Accuracy:  0.987121
Epoch 10/100: 100%|██████████| 613/613 [54:54<00:00,  3.74s/batches]
Epoch 11/100:  16%|█▋        | 101/613 [10:54<3:27:37, 24.33s/batches]
Improved Validation Accuracy:  0.988396
Epoch 11/100:  33%|███▎      | 201/613 [19:35<2:43:16, 23.78s/batches]
Improved Validation Accuracy:  0.988651
Epoch 11/100:  82%|████████▏ | 501/613 [46:34<47:10, 25.27s/batches]
Improved Validation Accuracy:  0.989926
Epoch 11/100: 100%|██████████| 613/613 [56:38<00:00,  4.01s/batches]
Epoch 12/100:  74%|███████▎  | 451/613 [42:36<1:10:29, 26.11s/batches]
Improved Validation Accuracy:  0.990946
Epoch 12/100: 100%|██████████| 613/613 [57:15<00:00,  4.12s/batches]
Epoch 13/100:  57%|█████▋    | 351/613 [34:47<1:51:28, 25.53s/batches]
Improved Validation Accuracy:  0.991201
Epoch 13/100:  65%|██████▌   | 401/613 [39:24<1:33:24, 26.44s/batches]
Improved Validation Accuracy:  0.991329
Epoch 13/100:  74%|███████▎  | 451/613 [44:00<1:10:48, 26.23s/batches]
Improved Validation Accuracy:  0.992221
Epoch 13/100: 100%|██████████| 613/613 [58:49<00:00,  3.92s/batches]
Epoch 14/100:  25%|██▍       | 151/613 [17:03<3:44:30, 29.16s/batches]
Improved Validation Accuracy:  0.992349
Epoch 14/100:  41%|████      | 251/613 [26:23<2:39:24, 26.42s/batches]
Improved Validation Accuracy:  0.992604
Epoch 14/100:  49%|████▉     | 301/613 [30:55<2:17:34, 26.46s/batches]
Improved Validation Accuracy:  0.992859
Epoch 14/100:  90%|████████▉ | 551/613 [53:20<26:33, 25.70s/batches]
Improved Validation Accuracy:  0.992987
Epoch 14/100: 100%|██████████| 613/613 [58:28<00:00,  3.75s/batches]
Epoch 15/100:  49%|████▉     | 301/613 [29:33<2:13:29, 25.67s/batches]
Improved Validation Accuracy:  0.993114
Epoch 15/100:  82%|████████▏ | 501/613 [47:05<44:50, 24.02s/batches]
Improved Validation Accuracy:  0.993752
Epoch 15/100: 100%|██████████| 613/613 [56:36<00:00,  3.76s/batches]
Epoch 16/100:   0%|          | 1/613 [02:59<30:30:04, 179.42s/batches]
Improved Validation Accuracy:  0.994134
Epoch 16/100:  74%|███████▎  | 451/613 [43:14<1:10:23, 26.07s/batches]
Improved Validation Accuracy:  0.994389
Epoch 16/100: 100%|██████████| 613/613 [57:17<00:00,  3.76s/batches]
Epoch 17/100:   8%|▊         | 51/613 [06:43<4:02:24, 25.88s/batches]
Improved Validation Accuracy:  0.994772
Epoch 17/100: 100%|██████████| 613/613 [56:03<00:00,  3.76s/batches]
Epoch 18/100:   0%|          | 1/613 [02:14<22:53:37, 134.67s/batches]
Improved Validation Accuracy:  0.995537
Epoch 18/100: 100%|██████████| 613/613 [56:51<00:00,  3.81s/batches]
Epoch 19/100:  49%|████▉     | 301/613 [29:22<2:14:38, 25.89s/batches]
Improved Validation Accuracy:  0.995664
Epoch 19/100:  98%|█████████▊| 601/613 [56:58<05:43, 28.60s/batches]
Improved Validation Accuracy:  0.99643
Epoch 19/100: 100%|██████████| 613/613 [57:45<00:00,  3.93s/batches]
Epoch 20/100: 100%|██████████| 613/613 [1:01:26<00:00,  3.82s/batches]
Epoch 21/100:  16%|█▋        | 101/613 [26:14<3:19:43, 23.41s/batches]
Improved Validation Accuracy:  0.996684
Epoch 21/100:  92%|█████████▏| 567/613 [1:09:40<08:29, 11.08s/batches]
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-10-4ae93ce00b29> in <module>()
     43 
     44             # Run optimization op (backprop) and cost op (to get loss value)
---> 45             _ = sess.run(optimizer, feed_dict={x: batch_X, y: batch_y, keep_prob: 0.5})
     46 
     47             total_iterations += 1

/Users/codemanneil/anaconda/envs/traffic-signs/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    715     try:
    716       result = self._run(None, fetches, feed_dict, options_ptr,
--> 717                          run_metadata_ptr)
    718       if run_metadata:
    719         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/codemanneil/anaconda/envs/traffic-signs/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    913     if final_fetches or final_targets:
    914       results = self._do_run(handle, final_targets, final_fetches,
--> 915                              feed_dict_string, options, run_metadata)
    916     else:
    917       results = []

/Users/codemanneil/anaconda/envs/traffic-signs/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
    963     if handle is None:
    964       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
--> 965                            target_list, options, run_metadata)
    966     else:
    967       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/Users/codemanneil/anaconda/envs/traffic-signs/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
    970   def _do_call(self, fn, *args):
    971     try:
--> 972       return fn(*args)
    973     except errors.OpError as e:
    974       message = compat.as_text(e.message)

/Users/codemanneil/anaconda/envs/traffic-signs/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
    952         return tf_session.TF_Run(session, options,
    953                                  feed_dict, fetch_list, target_list,
--> 954                                  status, run_metadata)
    955 
    956     def _prun_fn(session, handle, feed_dict, fetch_list):

KeyboardInterrupt: 
In [7]:
# Class used to save and/or restore Tensor Variables
saver = tf.train.Saver()
save_path = "model/best_model"

with tf.Session() as sess:
    # Load the weights and bias
    saver.restore(sess, save_path)
    test_accuracy = sess.run(accuracy, feed_dict=test_feed_dict)
    print("Test Accuracy: ", test_accuracy)
    
Test Accuracy:  0.958591

Best Validation Accuracy: 0.996684 Test Accuracy: 0.958591

Question 1

Describe the techniques used to preprocess the data.

Answer: I tested 3 methods for normalizing the images:

1. Compute mean/stddev per pixel and channel across the entire batch.  Subtract mean, divide by stddev for each pixel/channel.
2. Compute mean/stddev per image.  Subtract mean, divide by stddev for each  pixel/channel.
3. Convert each image to grayscale, perform min/max normalization, so each pixel has intensity in [-0.5, 0.5].

I was able to implement each of these methods efficiently, and didn't see a large difference in performance on a small subset I tested on, except a very small improvement on method 2.

I couldn't conclude that method 2 would actually perform better on the overall training set, but given my limited computing power, I decided to go forward with method 2 for actual training.

Looking back, I would test adding CLAHE processing to normalize brightness.

Question 2

Describe how you set up the training, validation and testing data for your model. If you generated additional data, why?

Answer:

The test set was initially separate on load, so I left that alone. The training data was split 80/20 into a training and validation set. Prior to data augmentation, I had an approximate 60/15/25 split between training, validation, and test sets respectively. Note that this prior to data augmentation, so the training set is higher after accounting for that.

For data augmentation, I only augmented the training data set. For obvious reasons I didn't want to touch the test set in any way. As for the validation set, I didn't want to perform data augmentation prior to the training-validation set split because that would have leaked validation data into the training set. I also didn't see a benefit of adding validation augmentation after the training-validation split, as I wanted the validation set to model the test set as closely as possible. For creating the augmented data, I used only random rotations of the image (between -45 and 45 degrees). I had also considered adding horizontal flips, but given that there were left and right turn signs, this seemed ill advised.

Looking back, I would likely add zooming (in and out) and skewing of the images for data augmentation. Another consideration would be to add different brightnesses of the images as augmented data.

Data Breakdown:

Number of training examples = 156835 Number of validation examples = 7842 Number of testing examples = 12630

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

Model

  1. Input [32 x 32 x 3] - Weight []
  2. Conv 1a [32 x 32 x 64] - Weight - [3 x 3 x 3 x 64]
  3. Conv 1b [32 x 32 x 64] - Weight - [3 x 3 x 64 x 64]
  4. Maxpool [16 x 16 x 64] - Weight - []
  5. Conv 2a [16 x 16 x 128] - Weight - [3 x 3 x 64 x 128]
  6. Conv 2b [16 x 16 x 128] - Weight - [3 x 3 x 128 x 128]
  7. Maxpool [8 x 8 x 128] - Weight - []
  8. Conv 3a [8 x 8 x 256] - Weight - [3 x 3 x 128 x 256]
  9. Conv 3b [8 x 8 x 256] - Weight - [3 x 3 x 256 x 256]
  10. Maxpool [4 x 4 x 256] - Weight - []
  11. FC 1 [1 x 1 x 512] - Weight - [4 x 4 x 256 x 512]
  12. Out [1 x 1 x 43] - Weight - [512 x 43]

Total Weights = 3,263,680

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

  • Optimizer: I used the Adam Optimizer. I attempted to use the standard gradient descent optimizer, but had issues with getting the learning rate at an effective level, and saw instant improvements in training rate when I swithced to Adam.
  • Batch size: I toyed with 64, 128, and 256, and saw the best results with 256.
  • Learning Rate: I started with a small initial learning rate for the Adam optimizer of 0.0001.
  • Epochs: I set the number of training epochs to 1000 (extremely high), because I also implemented an early stopping function that would stop if the validation accuracy hadn't increased over the last 1000 batches.

Question 5

What approach did you take in coming up with a solution to this problem?

Answer: I broke the project down into phases:

  1. Preprocess, augment, split, and save data to be quickly loaded.
  2. Create a simple training model, tune hyperparameters on a small subset of training data to get rough values, and test different data preprocessing techniques.
  3. Iteratively increase complexity of the model and train to early stopping. Compare validation accuracies of each model.

I used VGG Net as inspiration for my model, using multiple conv layers prior to pooling, in a pyramid shape. When it reached it's current form, it took ~14 hours to train on my laptop, and I'd reached over 99.5% validation accuracy. Adding any more complexity would have been computationally prohibitive, so I moved forward with final testing, which reached > 95% accuracy.


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

In [8]:
import csv

def get_sign_names():
    sign_names = dict()
    with open('signnames.csv', 'r') as csvfile:
        reader = csv.reader(csvfile)
        next(reader)   # skip first line
        for row in reader:
            sign_names[int(row[0])] = row[1]
    return sign_names

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It would be helpful to plot the images in the notebook.

Answer:

In [9]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
import cv2
import glob

def load_images_and_labels(folder, plot=False):

    new_image_path = folder + "/*.jpg"
    new_images = []
    for img in glob.glob(new_image_path):
        new_images.append(cv2.imread(img))
        
    # Convert BGR to RGB and resize image to 32x32x3 and plot image
    resized_images = []
    for img in new_images:
        b,g,r = cv2.split(img)              # get b,g,r
        rgb_img = cv2.merge([r,g,b])        # switch it to rgb
        resized_img = cv2.resize(rgb_img, (32, 32), interpolation=cv2.INTER_AREA)
        if plot:
            plt.imshow(resized_img)
            plt.show()
        resized_images.append(resized_img)
    
    # Load Labels
    with open(folder + '/labels.csv') as f:
        label_reader = csv.reader(f)
        row = next(label_reader)
        labels = [int(label) for label in row]
    
    return resized_images, labels
In [10]:
from sklearn.preprocessing import LabelBinarizer

def one_hot_encode(labels):
    encoder = LabelBinarizer()
    encoder.fit([i for i in range(43)])
    return encoder.transform(labels)
In [11]:
sign_names = get_sign_names()
X_signs_unprocessed, labels = load_images_and_labels('new_signs', plot=True)
X_signs = normalize_dataset_alt(X_signs_unprocessed)
y_signs = one_hot_encode(labels)

I tried to choose images that were photos of signs instead of just being signs on a white background. This should show how well the model predicts signs it would expect to see in the real world. Some other differences that may cause issues are:

1. The Speed Limit sign is from North America, so it's square instead of round.
2. The No Passing sign is slightly obscured by vegetation.
3. The deer on the Wild Animal Crossing sign is flipped horizontally from what is seen in the training set.

Question 7

Is your model able to perform equally well on captured pictures when compared to testing on the dataset?

Answer:

On the 5 images I pulled from online, I achieved an accuracy rate of 80%. Given the small sample size, and the fact that the second highest probability classification was correct on the missed sign, it's hard to say that the model performed equally well. Given some of the differences however between the new signs and the existing sign data (e.g. 30 mph vs 30 km/h, deer was flipped on wild animal sign) the model performed quite well.

In [26]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
save_path = "model/best_model"
saver = tf.train.Saver()
top_5_predictions = tf.nn.top_k(prediction, k=5)

def get_accuracy_on_new_images(X_signs, y_signs):

    with tf.Session() as sess:
        sess.run(init)
        # Load the weights and bias
        saver.restore(sess, save_path)

        # Stop Sign
        return sess.run(accuracy, feed_dict={x: X_signs, y: y_signs, keep_prob: 1.0})
    
In [27]:
print(get_accuracy_on_new_images(X_signs, y_signs))    
0.8

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

Answer:

The model is able to confidently predict theh stop sign, 30 mph (even though it was trained for 30 km/h), the left turn ahead sign, and the no passing sign. The one sign it failed to correctly predict was the Wild Animal Crossing sign, which was the 2nd place prediction. The wild animal crossing sign I downloaded was flipped horizontally compared to the training sign.

One thing I noticed is that scaling down the signs to 32x32 significantly blurred the defining features of the sign. It seems that if we had trained on higher resolution images and test images, we likely would have had significantly better results, though it also would have required a more complex and more difficult to train model.

In [28]:
def plot_results(sign_num, sign_indices, sign_values, sign_names, X_signs_unprocessed):
    # Sign Name
    print("Sign: ", sign_names[labels[sign_num]])
        
    # Plot Graph
    plt.subplot(1,2,1)
    top_sign_names = [sign_names[ind] for ind in sign_indices]
    y_pos = np.arange(len(top_sign_names))
    plt.barh(y_pos, sign_values, align='center', alpha=0.4)
    plt.gca().invert_yaxis()
    plt.yticks(y_pos, top_sign_names)
    plt.xlabel('Confidence')
    
    # Plot Image
    plt.subplot(1,2,2)
    plt.imshow(X_signs_unprocessed[sign_num])
    
    plt.show()
    print("\n")

def show_top_predictions(X_signs, y_signs, X_signs_unprocessed, sign_names): 
    saver = tf.train.Saver()
    with tf.Session() as sess:
        # Load the weights and bias
        saver.restore(sess, save_path)
        values, indices = sess.run(top_5_predictions, feed_dict={x: X_signs, y: y_signs, keep_prob: 1.0})
        for sign_num in range(len(values)):
            plot_results(sign_num, indices[sign_num], values[sign_num], sign_names, X_signs_unprocessed)
In [29]:
        
show_top_predictions(X_signs, y_signs, X_signs_unprocessed, sign_names)
Sign:  Wild animals crossing

Sign:  Turn left ahead

Sign:  No passing

Sign:  Speed limit (30km/h)

Sign:  Stop

Question 9

If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.

Answer: Run the code in the "Model" section to define the model as well as the "Data Processing" section to define the normalization methods used.

Put jpg images in a folder of your choice, along with a file called 'labels.csv'. Labels.csv should be the classId labels, on a single line, separated by commas, in alphabetical order based upon the image filenames.

Pass the path to this folder to the method defined below.

In [30]:
def classify_new_images(path):
    sign_names = get_sign_names()
    X_signs_unprocessed, labels = load_images_and_labels(path)
    X_signs = normalize_dataset_alt(X_signs_unprocessed)
    y_signs = one_hot_encode(labels)
    
    print("Accuracy: ", get_accuracy_on_new_images(X_signs, y_signs))
    
    show_top_predictions(X_signs, y_signs, X_signs_unprocessed, sign_names)
    
classify_new_images('new_signs')
Accuracy:  0.8
Sign:  Wild animals crossing

Sign:  Turn left ahead

Sign:  No passing

Sign:  Speed limit (30km/h)

Sign:  Stop

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In [ ]: